If we want to add metrics to the clients/servers then we should leverage middleware as it will be much cleaner and keep the business logic free from this instrumentation.

You can take a look at MystenLabs/sui-rust-sdk#237 for how middleware can be added to grpc clients and https://github.com/bmwill/sui/blob/bf4f4c2e66964ba3668baa2accbf0521c9aa0e81/crates/sui-rpc-api/src/lib.rs#L229-L243 for how middleware can be added to the server side (I think we do already do some server side metrics using this paradigm.)

Reverted the client.rs changes. Byte counting now lives in RpcP2PChannel which is the metrics wrapper around the client, so grpc::Client stays clean.

I looked at both references. The challenge with applying CallbackLayer on the client side for our use case: Client instances are cached in CommitteeSet and shared across protocols. The middleware wouldn't know which protocol is active, so we'd lose protocol labels on the bytes metrics. We need per-protocol labels to compare communication cost across protocols.

RpcP2PChannel is created per-protocol with the label, so it's the natural place for this.